Method for using a motorized camera mount for tracking in augmented reality

ABSTRACT

The invention is a method for displaying otherwise unseen objects and other data using augmented reality (the mixing of real view with computer generated imagery). The method uses a motorized camera mount that can report the position of a camera on that mount back to a computer. With knowledge of where the camera is looking, and the size of its field of view, the computer can precisely overlay computer-generated imagery onto the video image produced by the camera. The method may be used to present to a user such items as existing weather conditions, hazards, or other data, and presents this information to the user by combining the computer generated images with the user&#39;s real environment. These images are presented in such a way as to display relevant location and properties of the object to the system user. The primary intended applications are as navigation aids for air traffic controllers and pilots in training and operations, and use with emergency first responder training and operations to view and avoid/alleviate hazardous material situations, however the system can be used to display any imagery that needs to correspond to locations in the real world.

CROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims priority of Provisional patentapplication 60/349,028 filed Jan. 15, 2002. This application is acontinuation in part of patent application Ser. No. 09/634,203, filed onAug. 9, 2000.

FIELD OF THE INVENTION

[0002] This invention relates to computer graphics and augmentedreality. It defines a way to augment real imagery with a graphicalrepresentation of an object (in its actual location) in order tofacilitate completion of a task by a human.

COPYRIGHT INFORMATION

[0003] A portion of the disclosure of this patent document containsmaterial that is subject to copyright protection. The copyright ownerhas no objection to the facsimile reproduction by anyone of the patentdocument or the patent disclosure as it appears in the Patent andTrademark Office records but otherwise reserves all copyright workswhatsoever.

BACKGROUND OF THE INVENTION

[0004] This invention can be used any time a user needs to “see” anotherwise invisible object placed within the setting in which it wouldnormally appear, and in this case, onto a video stream. This form ofimage “augmentation” is known as augmented reality (AR), and in thiscase is considered video-based AR since it is only a video camera, asopposed to a see-through setup (as might be done with partial mirrors).A video-based AR setup has only a few basic components, including (1) acamera with a known field of view (FOV), (2) a method to determine theposition and orientation of the camera, and (3) a computer that createscomputer-generated virtual objects that correspond to what the camera isseeing by using the previous two components.

[0005] Typically, one of the physically largest components is the secondone, wherein a tracking system is used to determine the position andorientation of the camera. The invention creates an AR setup by using asimple, compact, inexpensive, integrated tracking system combined with avideo camera in one small unit. Such a camera and tracking combinationdevice is the EVI-D30 Motorized Camera, from Sony.

[0006] A common method known in the art would use an external trackingsystem available from a company like InterSense (Burlington, Mass.),where the tracking system is made of two main parts. One part is arelatively substantial device that attaches to the camera, and the otherpart is a detection system that must be supported in place above thecamera, usually by at least a few feet. While it has the advantage ofbeing able to detect translational motion in addition to orientation, itis substantially larger than an integrated solution.

SUMMARY OF THE INVENTION

[0007] When using a video camera, it is sometimes useful to superimposecomputer-generated graphics onto its view. To accomplish that, thisinvention uses a video camera with a built-in motor and position encoderto both control and know its position. By using this method, thecomputer has knowledge of where the camera is looking, thereby allowingthe computer to superimpose graphical elements representing realobjects, anchored in the real world, onto the view of the camera.

[0008] This method of tracking using a motorized camera mount can beadapted to a wide variety of situations. The sports entertainmentindustry could, for example, use this invention to train a cameraoperator to follow an object which can be difficult to see (such as agolf ball or hockey puck). The camera operator will be better preparedto film the action and the viewers will receive a more complete accountof the event.

[0009] This invention facilitates the blending of computer-generated(virtual) graphical elements (which represent real world objects) withthe real-world view of the user. The motorized camera mount in thisinvention is used to track the location of the user and of the actualobject so that the virtual (computer-generated) imagery can be placed inthe same location in which the object would appear if it were actuallyvisible. Superimposing virtual imagery onto a real world scene (whichmay be static or changing) and presenting the combined image as the“replacement” for the real world image is known as Augmented Reality(AR). The location and properties of the computer generated graphicalelements are known and may be input via a human or other means. Theuser's real world scene/image might consist of scenery that the user cannormally see.

[0010] The user will visualize the combined virtual and real image usingtraditional interfaces such as monitors, or the data may be projectedinto a Head-Mounted Display (HMD). In any case, the real image (e.g., anairfield or airway, along with accompanying structures and aircraft),will be shown, overlaid with computer-generated graphical elements. Thenet result is an augmented reality.

[0011] The invention features a method for using a motorized cameramount for tracking in Augmented Reality. Real world imagery (captured byvideo or other method) is input into a system and augmented withcomputer-generated imagery of otherwise unseen objects and presented tothe user as the replacement for the real world image.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012]FIG. 1 is a schematic view of a motorized camera and motorizedmount connected to a computer for the purpose of tracking and videocapture for augmented reality, for use in the preferred embodiment ofthe invention.

[0013]FIG. 2 is a close-up view of the camera and motorized mount ofFIG. 1.

[0014]FIG. 3 schematically depicts an augmented reality display withcomputer-generated indicators displayed over an image as an example of aresult of this invention.

[0015]FIG. 4 is the un-augmented scene from FIG. 3 withoutcomputer-generated indicators. This image is a real-world image captureddirectly from the camera.

[0016]FIG. 5 is an augmented reality display of the same scene as thatof FIG. 3 but from a different camera angle where the computer-generatedindicators that were in FIG. 3 remain anchored to the real-world image.

[0017]FIG. 6 is the un-augmented scene from FIG. 5 withoutcomputer-generated indicators.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS OF THE INVENTION

[0018] The preferred embodiment of the invention utilizes a motorizedcamera mount with a built-in position tracker. The properties of thecomputer-generated graphical elements are determined by a independentsource (e.g., the user, sensor information, or other method of input).The method uses augmented reality (the mixing of real media withcomputer generated media) to present this information in a format thatcombines the computer-generated images with the user's real environment.The user then visualizes the combined virtual and real image via atraditional interface such as a computer monitor, or via another method,such as a Head-Mounted Display (HMD).

[0019] The captured video image of the real world is mixed with thecomputer-generated graphical elements via an onboard or external imagecombiner to form an augmented reality display. Onboard mixing isperformed via software. External mixing can be provided bycommercial-off-the-shelf (COTS) mixing hardware, such as a Videonicsvideo mixer or Coriogen Eclipse keyer. Such an external solution wouldaccept the video signal from the camera and a computer generated videosignal from the computer and combine them into the final augmentedreality image.

[0020] This invention can be used operationally to show a personperforming a task the location of an object that would otherwise not bevisible to the video camera. Situations where it may be advantageous torepresent items in this manner include: (A) a runway/runway lights thatcan't be seen in a heavy fog by a pilot (here the plane would have otherhardware that would determine the location of the runway relative to thepresent orientation of the plane, and then the computer could augment aview showing this), (B) invisible gases or other dangers (here, sensorscould detect the presence and flow of the gases and then display it onthe camera to show the user where the gases are in the field of view ofthe camera), (C) an airport runway controller (or anyone) may want to beable to display the locations of vehicles or other moving objects thatare difficult to see in the field of view of the camera, but whoselocations are known (for example, at an airport with heavy fog, vehicles(trucks or even planes) could be moving at the airport, and separatesensors (perhaps a radar system, or beacons that transmit back thelocation of the vehicle) determine the positions of the vehicles forproper display to the viewer).

[0021] The invention may also be used in a training scenario. Thisincludes any situation where it would be advantageous to train someonehow to do a task by displaying virtual objects onto the view of acamera. For example, training a cameraman to track golf balls at a golftournament could be accomplished faster and more effectively using theinvention.

[0022] The base of the camera may, or may not be stationary. If the baseis not stationary, the moving base must be tracked by a separate 6DOFmethod. This situation could be applicable on a ship, airplane, orautomobile where the base of the camera mount is fixed to the movingplatform, but not fixed in world coordinates. A GPS tracking system, anoptical tracking system, or some other kind of tracking system mustprovide a position and orientation of the base of the camera. Forexample, a GPS system could be used to find the position and orientationof the base. It would then use the camera's orientation sensors todetermine the camera's orientation relative to the camera's base, theorientation and position of which must be known. Such a system could beplaced on a vehicle, aircraft, or ship. Another example would includemounting the camera base on a 6DOF gimbaled arm. As the arm moves, itcan be mechanically tracked in 3D space. Similar to the previousexample, this position and orientation can be added to the data from thecamera to find the camera's true position and orientation in worldcoordinates.

[0023] The invention may also use an open-loop architecture, in whichthe computer cannot request a report from the camera containing currentorientation data. In this case, the computer drives the camera mount toa specified orientation, and external motion of the camera is notpermitted. In such an implementation, the system knows the position ofthe camera by assuming that the camera, in fact, went to the lastlocation directed by the computer. Similarly, the system may also use afeedback architecture. In this scenario, the system will send a commandto the camera to move to a specified position, and then the system mayrequest a report from the camera that contains the current position ofthe camera, correcting it again if necessary.

[0024] Finally, the system may operate in a calibrated configuration, inwhich a computer-generated infinite horizon and center-of-screenindicator are used to verify anchoring and registration ofcomputer-generated objects to real-world positions. In this case, thecomputer can know exactly where the camera is looking in fully correct,real world coordinates. The system may also operate in an uncalibratedconfiguration, which would not guarantee perfect registration andanchoring but which may be suitable in certain lower-accuracyapplications.

[0025]FIG. 1 illustrates the hardware for the preferred method of theinvention. A motorized video camera (30, 31) is used as a trackingsystem for augmented reality. By connecting the motorized video camerato the computer (20) via an RS-232 serial cable (33) (for camera controland feedback) and video cable (32), the camera may be aimed, theposition of the camera can be queried, and the image seen by the cameramay be captured over the video cable (32) by software running on thecomputer. Additionally, the computer can query the camera for itscurrent field of view, a necessary piece of information if the computerimage is to be rendered properly.

[0026]FIG. 2 is a close-up view of the preferred Sony EVI-D30 motorizedcamera. The camera is composed of a head (30) and a base (31) coupled bya motorized mount. This mount can be panned and tilted via commands fromthe computer system, which allows the head to move while the baseremains stationary. The camera also has internal software, which tracksthe current known pan and tilt position of the head with respect to thebase, which may be queried over the RS-232 serial cable.

[0027] The video signal from the camera travels into a video capture, or“frame grabber” device connected to the computer. In this embodiment ofthe invention, an iRez USB Live! capture device is used, which allowssoftware on the computer to capture, modify, and display the image onthe screen of the computer. This image source can be combined withcomputer-generated elements before display, allowing for augmentedreality applications.

[0028] In FIG. 3, an augmented reality display using the EVI-D30 as atracked image source is shown. This image is a composite imageoriginally acquired from the camera, which is displayed in FIG. 4, andshows furniture and other items physically located in real space (40)(50) (60). The software running on the computer then queries the camerafor its orientation. The orientation returned from the camera representsthe angle of the camera's optics with respect to the base of the camera.By corresponding this information with the known location andorientation of the camera base, a real-world position and orientationcan be computed for the camera's optics. These data are then used torender three-dimensional computer-generated poles (70) with properperspective and screen location, which are superimposed over the imagecaptured from the camera. The resulting composite image is displayed tothe user on the screen.

[0029]FIG. 5 shows the same scene as FIG. 3, but from a different angle.The unaugmented version of FIG. 5 (shown in FIG. 6) is captured from thevideo camera, and the computer-generated elements (70) are again addedto the image before display to the user. Note, as the camera angle haschanged, the perspective and view angle of the poles (70) has alsochanged, permitting them to remain anchored to locations in thereal-world image.

[0030] The preferred embodiment of this invention can be reduced topractice using the hardware and software listed in Table 1. TABLE 1Equipment List Item Brand Name EVI-D30 Motorized Camera SONY PowerSupply (for EVI-D30) SONY Inspiron 8200 Laptop Computer (“Pegasus”) DELLCable (8-pin Mini-DIN VISCA ™ to SONY 9-pin D-SUB RS-232) Cable(S-Video) USB Live! Video Capture Device iRez Visual C++ 6.0 (CD)MICROSOFT

What is claimed is:
 1. A method for using a motorized camera mount as atracking system for Augmented Reality (AR) comprising: capturing animage or view of the real world; determining the orientation of a camerabeing carried by the camera mount by obtaining information from themotorized camera mount; using a computer to generate a graphical imagerepresenting unseen information that corresponds to the knownorientation of the viewpoint of the camera; augmenting a view of thereal world with the computer generated image; and presenting theaugmented view to the user.
 2. The method of claim 1 wherein theaugmenting step comprises using onboard video mixing through use of avideo capture device with the computer, and the capturing step comprisescapturing a video of reality.
 3. The method of claim 1 wherein theaugmenting step comprises using an external video mixing solution, tocombine real and computer-generated graphical elements outside of thecomputer.
 4. The method of claim 1 for use in operations.
 5. The methodof claim 1 for use in training.
 6. The method of claim 1 in which thedetermining step comprises calibrating the camera and camera mount. 7.The method of claim 1 in which the camera mount is coupled to a fixedplatform.
 8. The method of claim 1 where the determining step comprisesusing the camera and camera mount in conjunction with a separatetracking system to generate a combined position and orientation value.9. The method of claim 1 in which the determining step comprises usingthe camera and camera mount and using the computer to request thecurrent camera position to thus utilize a feedback architecture.
 10. Themethod of claim 1 in which the determining step comprises using thecamera and camera mount and using a feed-forward architecture.
 11. Themethod of claim 1 in which the camera mount is not stationary.
 12. Themethod of claim 11 in which the camera mount is attached to a vehicle.13. The method of claim 11 in which the camera mount is attached to anaircraft.
 14. The method of claim 11 in which the camera mount isattached to a watercraft or ship.
 15. The method of claim 11 in whichthe camera mount is attached to a gimbaled arm.
 16. The method of claim1 in which the determining step comprises the motorized camera mountreporting the field of view of the camera to the computer.